Association between quality of clinical practice guidelines and citations given to their references

It has been suggested that bibliometric analysis of different document types may reveal new aspects of research performance. In medical research a number of study types play different roles in the research process and it has been shown, that the evidence-level of study types is associated with varying citation rates. This study focuses on clinical practice guidelines, which are supposed to gather the highest evidence on a given topic to give the best possible recommendation for practitioners. The quality of clinical practice guidelines, measured using the AGREE score, is compared to the citations given to the references used in these guidelines, as it is hypothesised, that better guidelines are based on higher cited references. AGREE scores are gathered from reviews of clinical practice guidelines on a number of diseases and treatments. Their references are collected from Web of Science and citation counts are normalised using the item-oriented zscore and the PPtop-10% indicators. A positive correlation between both citation indicators and the AGREE score of clinical practice guidelines is found. Some potential confounding factors are identified. While confounding cannot be excluded, results indicate low likelihood for the identified confounders. The results provide a new perspective to and application of citation analysis. Conference Topic Old and New Data Sources for Scientometric Studies: Coverage, Accuracy and Reliability (Topic 2) and Science Policy and Research Evaluation: Quantitative and Qualitative Approaches (Topic 3) Introduction While most scientometric studies of research publications focus on standard journal articles, it has been suggested several times that other document types may play a role in research assessment as well (Lewison, 2002, 2003; van Leeuwen, Costas, Calero-Medina, & Visser, 2012). Lewison (2002, 2003) in particular has emphasised the possibilities of various document types in the medical fields. One document type particular to that field is the clinical practice guideline. The purpose of these guidelines is to gather the best evidence of the treatment of diseases to ensure the best possible treatment at hospitals, clinics and general practices (The AGREE Collaboration, 2003). One might therefore expect these guidelines to build on the highest quality research available. Studies have shown a connection between citation scores and the evidence-level of clinical study types (e.g. Andersen & Schneider, 2011; Kjaergard & Gluud, 2002; Patsopoulos, Analatos, & Ioannidis, 2005). These study types are hierarchically ordered, assigning greater importance to the evidence found in high-level studies, such as meta-analyses and randomised, controlled trials (RCT), than lower level studies, such as case studies (Greenhalgh, 2010). This hierarchy is widely applied in different areas of health research, and the connection between citations and evidence levels indicates that high-evidence studies are indeed, on average, used more than other. We might thus speculate if not references used in clinical practice guidelines are cited more on average than other papers, if the guidelines indeed represent the best available evidence on a topic. Several studies have indicated that clinical practice guidelines are created very differently, with great variation in scope, rigor, clinical recommendation and overall quality (e.g. Burda, Norris, Holmer, Ogden, & Smith, 2011; Ferket et al., 2010, 2011; Freel et al., 2008; Gallardo et al., 2010; Kis et al., 2010). Many of these reviews of clinical practice guidelines use the AGREE 1 instrument (The AGREE Collaboration, 2003) to assess six major aspects of guideline development. Especially one aspect, rigor of development, is directly related to the evidence found in the background literature. The hypothesis presented here is that there is a positive correlation between the rigor of development AGREE score of clinical practice guidelines and the citations given to the literature references in these clinical practice guidelines. If this correlation can be observed it points at an association between clinical evidence, citations and the development of clinical practice guidelines. The association cannot be assumed to be causative, however, but would still provide valuable insights into the inclusion and interpretation of a new document type in research assessment. Paper outline The following section outlines the acquisition of data, from reviews of clinical guidelines (top-level) to the actual guidelines and their references. This is followed by a presentation of the citation indicators used to assess the citation impact of the references of the guidelines. These references will also be discussed further in the results section which begins with an analysis of the citations given to the references, to test if they are representative of a standard citation distribution. The section concludes with a correlation analysis of the tested indicators versus the AGREE scores. All results are discussed in the final section and known weaknesses of this study are presented and discussed with respect to the findings. Materials and Methods The AGREE instrument consists of 23 key items organised in six domains, with the intention of describing various aspects of guideline development. The six domains are scope and purpose; stakeholder involvement; rigor of development; clarity and presentation; applicability; and editorial independence (The AGREE Collaboration, 2003). These domain 1 Appraisal of Guidelines for Research and Evaluation N = 3 AGREE scores not reported as percentage N = 5 Not a review of disease or surgery guidelines. Dentistry (N = 1) also excluded. N = 57 AGREE scores not reported, or only as cumulative scores. N = 147 Clinical practice guidelines not retrievable from Web of Science, using cited reference search N = 4 Reference lists with no items